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Distributed Multilevel Diversity Coding 

Zhiqing Xiao, Jun Chen, Yunzhou Li, and Jing Wang 


Abstract 

In distributed multilevel diversity coding, K correlated sources (each with K components) are 
encoded in a distributed manner such that, given the outputs from any a encoders, the decoder can 
reconstruct the hrst a components of each of the corresponding a sources. For this problem, the optimality 
of a multilayer Slepian-Wolf coding scheme based on binning and superposition is established when 
K < 3. The same conclusion is shown to hold for general K under a certain symmetry condition, which 
generalizes a celebrated result by Yeung and Zhang. 

Index Terms 

Data compression, diversity coding, entropy inequality, Lagrange multiplier, linear programming, rate 
region, Slepian-Wolf, superposition. 


I. Introduction 

Consider the scenario where K correlated sources Ui,U 2 , - ■ ■ , Uk are compressed by K sensors in 
a distributed manner and then forwarded to a fusion center for joint reconstruction. This is exactly the 
classical distributed data compression problem, for which Slepian-Wolf coding fill, lEl is known to be 
rate-optimal. However, to attain this best compression efficiency, encoding at each sensor is performed 
under the assumption that all the other sensors are functioning properly; as a consequence, inactivity of 
one or more sensors typically leads to a complete decoding failure at the fusion center. Alternatively, 
each sensor can compress its observation using conventional point-to-point data compression methods 
without capitalizing the correlation among different sources so that the maximum system robustness can 
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be achieved. In view of these two extreme cases, a natural question arises whether there exists a tradeoff 
between compression efficiency and system robustness in distributed data compression. 

One approach to realize this tradeoff is as follows. Specifically, we decompose each into K 
components Uk,i,Uk, 2 , ■ ■ ■ ,Uk,K, ordered according to their importance, and encode them in such a 
way that, given the outputs from any a sensors, the fusion center can reconstruct the first a compo¬ 
nents of each of the corresponding a sources. The aforedescribed two extreme cases correspond to 
(Uk,!,--- ,Uk,K-i,Uk,K) = (0, ••• ,0,Uk) and {Uk,i,Uk, 2 , ■ ■ ■ ,Uk,K) = (C/fc,0, ••• ,0), respectively. 
One can realize a flexible tradeoff between compression efficiency and sysfem robusfness by adjusting 
fhe amounf of information allocated to different components. We shall refer to this problem as distributed 
multilevel diversity coding (D-MLDC) since it reduces to the well-known (symmetrical) multilevel 
diversity coding (MLDC) problem when Ui^a = = • • • = UK,a almost surely for all a. 

The concept of MLDC was introduced by Roche ||3l and more formally by Yeung |@| though research on 
diversity coding can be traced back to Singleton’s work on maximum distance separable codes Q. The 
symmetric version of this problem has received particular attention 0, and arguably the culminating 
achievement of this line of research is the complete characterization of the admissible rate region of 
symmetrical MLDC by Yeung and Zhang Q. Some recent developments related to MLDC can be found 
in Hl-ini. 

The goal of the present paper to characterize the performance limits of D-MLDC, which, we hope, may 
provide some useful insights into the tradeoff between compression efficiency and system robustness in 
distributed data compression. More fundamentally, we aim to examine the principle of superposition 0| 
in the context of D-MLDC. Although superposition (or more generally, layering) is a common way 
to construct sophisticated schemes based on simple building blocks and often yields the best known 
achievability results, establishing the optimality of such constructions is rarely straightforward, especially 
when encoding is performed in a distributed manner. In fact, even for the centralized encoding setup 
studied in Q, the proof of the optimality of superposition is already highly non-trivial. This difficulty 
can be partly attributed to the fact that it is often a technically formidable task to extract layers from a 
generic scheme using information inequalities in a converse argument, even in cases where the use of 
layered constructions may appear rather natural. 

From this perspective, our work can be viewed as an initial step towards a better understanding of 
layered schemes for distributed compression of correlated sources. We shall propose a multilayer Slepian- 
Wolf coding scheme based on binning and superposition, and establish its optimality for D-MLDC when 
K < 3. This scheme is also shown to be optimal for general K under a certain symmetry condition, which 
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generalizes the aforementioned result by Yeung and Zhang on symmetrical MLDC JTl. The main technical 
difficulty encountered in our proof is that it appears to be infeasible to characterize the admissible rate 
region of D-MLDC by deriving inner and outer bounds separately and then making a direct comparison 
based on their explicit expressions. To circumvent this difficulty, we follow the approach in Q, where 
the analysis of the inner bound and that of the outer bound are conceptually intertwined. Specifically, 
we analyze cerfain linear programs associated with the achievable rate region of the proposed scheme 
and leverage the induced Lagrange multipliers to establish the entropy inequalities that are needed for 
a matching converse. Since the problem considered here is more general than that in Q, the relevant 
linear programs and entropy inequalities are inevitably more sophisticated. It is worth mentioning that, 
in a broad sense, the strategy of determining an information-theoretic limit by connecting achievability 
and converse results to a common optimization problem (not necessarily linear) via duality has find 
applications far beyond MLDC (see, e.g., HU). 

The rest of this paper is organized as follows. We state the basic definitions and the main results in 
Section ini Section ITm contains a high-level description of our general approach. The detailed proofs can 
be found in Sections |IV] and |Vl We conclude the paper in Section |Vll 

Notation: Random vectors {Xt,t E T) and ^T,t' £ T') are sometimes abbreviated as Xt 

and Xt,t', respectively. For two integers xi,X 2 E Z, we define [xi ■. X 2 ] = {x E Z : xi < x < X 2 }. 
The cardinalify of a finite set V is denoted by |1/|; moreover, for any T C [1 : |1/|], let (V}j^ = 
{v £ V : \{v' £V : v' < v}\ £ T}. We often do not distinguish between a singleton and its element. 


IT Problem Formulation and Main Results 


A. System Model 

Let k £ [1 : K], be K vector sources. We assumq^ that f7[i:X]^Q, a E [1 : K], are mutually 

independent whereas, for each a, the components of f7[i:X],a Uk,a, A: E [1 ; K]) can be arbitrarily 
correlated. Let be i.i.d. copies of U[i-,k],Ii-.k]- 

An (n, {Mk,k £ [1 ; K])) D-MLDC system consists of: 

• K encoders, where encoder Encfc (A: E [1 ; K]) maps the source sequence to a symbol Sk in 


'This assumption can be relaxed to a certain extent and can be modified in various ways. In this paper we do not seek to 
present our results in their most general forms since the resulting statements and expressions may become rather unwieldy. 
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[1 : Mfc], i.e., 

K 

Encfc: 

a=l 

^k,[l-.K] ^k, 

• 2^ — 1 decoders, where decoder Decy (0 ^ C [1 : K]) produces a reconstruction of f^y[i.|y|] 
denoted by ^^y[i.|y|], based on Sy, i-e., 

|V'| 

kGV keVa=l 

Sv ^!^y,[i:|y|]- 

A D-MLDC system with K = 3 is illustrated in Fig. [T] 



Tjn 

> ^3,2 

Tjn frn 

^ 2.15 ^ 2,2 
fjn 
» ^3,2 

fjn fjn frn 
^ 1 . 1 ’ ^ 1,25 ^ 1,3 
fjn fjn 

> ^ 2,25 *-^ 2,3 

fjn frn 

) U o n U o o 


Fig. 1. System diagram for D-MLDC with K = 3. 
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B. Admissible Rate Region 

A rate tuple {R/,, A: G [1 : AT]) is said to admissible if, for any e > 0, there exists an (n, {Mk,ke[l:K])) 
D-MLDC system such that 

(1) (Rate Constraints) 

- log Mfc < + e, k€[l:K], (1) 

n 

(2) (Reconstruction Constraints) 

Pr 7^ r?,|l:|v|l} < «, e 77 V' C |1 : if]. (2) 

The admissible rate region TZ’^ is defined as the set of all admissible rate tuples. 


C. Multilayer Slepian-Wolf Coding 

We shall propose a D-MLDC scheme, which can be viewed as a natural extension of that in Q to 
the distributed encoding setup. This scheme, termed multilayer Slepian-Wolf coding, includes two steps: 
intralayer coding and interlayer coding. 

• Intralayer Coding: For each a G [1 : AT], encoder /c (/c G [1 : AT]) compresses using the con¬ 
ventional binning schemelj of rate r^^a', correct reconstruction of (7^ a G 17, based on the cor¬ 
responding bin indices is ensured (with high probability) for all 17 C [1 : AT] with |17| = a if 
{rk,a, A: G [1 : AT]) G Ur, a, where 

nK,a = I (rA:,a, A: G [1 : iT]) : > H {Uv,a\Uv',a ), ^ G F' G I 


kev 


with 


yK,a = {V C[l:K] : 1 < |y| <«}, 

^K,a [^] = {V C [1 : iT] \c : |c'| + \V\ = a}, Ve YK,a- 

Interlayer Coding: In this step, encoder A: (A: G [1 : A']) generates its output by combining the bin 
indices associated with UJI^, a G [1 : A'], via superposition. Note that the resulting rate region 

{rk,a,ke [1 : K]) : {rk,a,ke [1 : K]) G a G [1 : AT]! 


I, a=l 

is an inner bound of TZr, i.e.. 


IZk C 7Z 


K- 


( 3 ) 


^Here one can in fact use universal Slepian-Wolf coding so that encoding and decoding can be performed without the 
knowledge of the source distribution (m. 
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D. Main Results 

Our first main result shows that TZk coincides with TZ*^^ when K < 3. 

Theorem 1: TT^ = TZk for K <3. 

To state our second main result, we need the following definition. 

Definition 1 (Symmetrical Source): We say that the distrihution of t7[i:X],[i:K] is symmetrical entropy- 
wise if H {Uv,a) = H {Uv',a) for all a e [I : K] and V, V C [1 : K] with |y| = |1/'|. 

It is worth noting that the symmetrical MLDC problem studied in Q corresponds to the special case 
where H {Uv,a) = H {Uv',a) for all a G [1 : K] and 0 C y, V' C [1 : K]. 

Theorem 2: If the distrihution of is symmetrical entropy-wise, then 7^^ = TZk- 

III. Outline of a General Approach 

In this section we attempt to give an outline of our general approach, which is in principle not restricted 
to the cases covered hy Theorems [T] and |2l On a conceptual level, this approach was originated in Q 
(and made more evident in IITOl ). It consists of three major steps: 

1) characterize the supporting hyperplanes of TZk (more precisely, the supporting hyperplanes of TZk, a, 
a G [1 : K]) via the analysis of the corresponding linear programs; 

2) establish a class of entropy inequalities based on the Lagrange multipliers induced by the aforemen¬ 
tioned linear programs; 

3) derive a tight outer bound on TZk by leveraging these entropy inequalities. 

A. Linear Program 

Each supporting hyperplane of TZk, a is associated with a linear program 

K 

„ : min '^Wkrk,a 
k=l 

over k G [1 : K] , 

S.t. J]rfc,a>i7(C/ua|C/y',a), V GYK,a,V' GY'K,a[V], 
kev 

where w = {wk,k G [1 : K]) G M+. It often suffices to consider the case where the weights Wk, k G 
[1 : K], are ordered. For this reason, we define 

Wk = Iw : rui > m2 > • • • > wk > wk+i — o| . 
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Moreover, to facilitate subsequent analysis, we introduce the following parti tioil^ of Wk for each a G 
[2:K]: 

= (w € Wk : TUi < ’ 

k=2 J 


f 1 ^ 1 ^ ] 

^Ka - S w G Wk : Wi > —— ^ Wk and wi+i < _ , ^ XI ^ G [1 : a - 2] , 

I “ ‘ k-l+l “ + k-l+2 ) 

\ /v-fV- / 


,(a-l) A 

'Km 


= I w G Wk : 'Wa-I > X 

I k=a ] 


We set W^Ji = Wk- 


Definition 2 (Lagrange Multiplier): We sa>^ (^cyit/qa;^ S ^K,a,V' G is an optimal La¬ 

grange multiplier of LP^q, with w G if 


E E Cv\V',aH{UvAUv',a)= (4) 

VeVK,„ y'ev'K,„[y] 

X X ^V\V',a =Wk, A: G [1 : -ff], (5) 

yeVK,„:fcey y'ev'K,„[y] 

cy|yq«>0, y g VK,a, -t"'€ V'k,„ [F] , 


where denotes the optimal value of LP^ 

It is in general not easy to find optimal solution A: G [1 : K]^ and optimal Lagrange multiplier 

(cviv',a, V G VK,a, G V'^,„ [V]) for a given LP^ ^ (see Section |IV] for a detailed analysis of LP^ 2 )- 
However, the task becomes relatively straightforward when a = 1 or a = iL as shown by the following 
two lemmas (which can be proved via direct verification). 

Lemma 1: For linear program LP^j with w G M+, ^ • -^l) optimal solution and 

(cy| 0 ,i)fo G ^K,\) is the unique optimal Lagrange multiplier, where 

T°^^^H{Uk,i), A:G[l:iF], 

C{fc}| 0 ,i =Wk, A; G [1 : iL]. 


^For w e Wk, we have wi > EXi > -^Y)k^v+iWk, G [1 : 1], and wi < ^ Ef=i+i 

rTi' < EfcLi'+i Wk, I' £ [I ■■ a — 1]. Therefore, • • • , indeed form a partition of Wk- 

"'it can he shown that (cv\v',a, V G Vx.a, V' G Y'K,a [L]) is an optimal Lagrange multiplier of LP^ ^ if and only if it is 
an optimal solution to the (asymmetric) dual problem of LP^ 
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Lemma 2: For linear program LP^ with w € Wx, /c € [1 : Ky is an optimal solution and 

{cv\[i-.K]\v,K^ ^ S ^k,k) is an optimal Lagrange multiplier, where 

A H {Uk,K\U[k+i:K],K) , ke[l:K], 

C[l:k]\[k+1:K],K — Wk - Wk+1, k € [I : K] , 
cv\[i:K]\v,K = 0 , Otherwise. 

The general case w € can he reduced to the case w € W/r via suitable relahelling. 


B. Entropy Inequality 

In this step we aim to establish a class of entropy inequalities needed for a matching converse by exploit¬ 
ing the properties of optimal Lagrange multipliers of LP^ a € [1 : iT]. More precisely, we shall identify 
suitable conditions under which there exist optimal Lagrange multipliers {cvlV',a,y a, V a H 
a € [1 : iT], such that 


E E Cv\V',a'H {Xv\Xv') > E E cviv',aH{Xv\Xv>) (6) 

V€VK,a.' [V] V&Yk.c [V] 

for all Xyi-x] and a > a'. 

The following lemma indicates that ® always holds when a' = 1. 

Lemma 3: Let (cy |0 i, "L G ^K,t) and {cy\yi^a^ V S VK,a, V' G ^ [L]^ be optimal Lagrange mul¬ 
tipliers of LP^]^ and LP^^,, respectively. We have 

E E E CvlV',akI {Xy\Xy>) 

V£Vk,i v^eVK.c V'ev'K,„[y] 

for all Xyi-xy 

Proof: According to (|5]), 


C{fc}|0,l — Wk — ^ ^ Cy|y/ Q,, /c G [1 : AT] . 

V&YK.c-k&V VeY'^^JV] 


( 7 ) 
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K 

Cy|0^ii? (Xv) = ^ C{fc}|0_ii7 (Xk) 

V£Wk,i k=l 

K 

= E E E Cv\V',aH{Xk) (8) 

k=l Ve¥K,a.:keV V'GVi^,„[y] 

= E E CV'|V'',o E H{Xk) 

fsVk.c y'ev'K,„[y] kev 

^ E E Cv\V',aH{Xv) 

veYK,c V'gy'k^JV] 

£ E E Cv\V',aH{Xv\Xv'), 

veYK,c V'ev'K,„[v] 

where ([8]l is due to ([7]). This completes the proof of Lemma [3] ■ 

C. Outer Bound 

As shown by the following lemma, the existence of entropy inequalities ® implies that TZk is an 
outer bound of 7^^. 

Lemma 4: If for any w G there exist optimal Lagrange multipliers (cy|y',Q,, V G YK,a, V' € q, 
a G [1 : iT], such that ® holds, then 

n*KcnK. (9) 

Proof: Let {Rk,k G [1 : iT]) be an arbitrary admissible rate tuple. It suffices fo show thaf 

K K 

Y.WkRk>Y.f"^ (10) 

k=l 0=1 

from which (|9l) follows immediately. We shall prove via induction that, for any D-MLDC system satisfying 
O and (111), 

K d 

Y.Wk{Rk + e)>Y,f^ + - Y. E Cy|y,^7f(5y|[/[U],[l:/3],V) 

k=i 0=1 veVK,3 V'ev'K,p[y] 

K 

/3e[l-.K], (11) 

k=l 

where Sg tends to zero as e —0. One can deduce (ITOl) from (fTTl) by setting jd = K and sending e —0. 
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It can be verified that 

K ^ K 

V] Wfc (i?fc + e) > - ^ Wk log Mfc 

k=l k=l 

1 ^ 

> - '^WkH{Sk) 
n 

k=l 

= ^ E <^v\%,iH{Sv), (12) 

^ V"GVk,i 

where (Y2\ is due to the fact (see Lemma [1]) that the optimal Lagrange multiplier (cy |0 i, V € Vk,!) is 
uniquely given by C{fc}| 0 ,i — Wk, /c G [1 : K]. Note that 


H {Sv) > H + H {Sv\U^,^) - H (C/{l,i|5y) 

> H + H {SvPl^ - n5, (13) 

> H {Uv,i) + H (Nv|[/[" - n6,, V G (14) 


where ([T3l) follows by ([ 2 ]) and Fano’s inequality. Substituting (fT4l) into (fT^ and invoking (lU) proves (fTTl) 
for 13 = 1. Now assume that (ITTI) holds for j] = B — 1. In view of ([hll, we have 

V'eVK,B-i V'ev'^^B.dV] 

> E E ( 15 ) 

VSNk.b y'GVk.sl^] 

It can be verified fhaf 


H 


"n c \ 

= H {u^,b,Sv\U^,..kui:B-i]^Sv') - H {u^,b\UI[.,kui:B-i], Sv, Sy) 

-n5„ V€Yk,b,V' 

where (O follows by dU) and Fano’s inequalify. Moreover, 


(16) 


H > H {ulsWlKUVB-lvSy) + H (^y|[/[" , Sy,) 

> H {K,b\K>,b) + H [Sv\Ull.K^^^,.B],Sv^ (17) 

= H {Uv,b\Uv',b) + H , (18) 
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where (fTTl) is due to the fact that Uy^ Uy, ^ i^\i-K] [i s]’ a Markov chain. Continuing 

from (fTSl) . 


E E Cv\V',B-l 

v&Yk.b-i V"'ev'K,_B_i[t^] 

E E cv\V',bH {Uv,b\Uv',b) 

vgWk.b V'ev'K,fl[v] 


+ E E ^v\V',bH (^Sv\U^yj^^^^y^^,Sv''^ 

V&Yk.b V'£V'j, s[V] 

— nSf: E E cv\V',B (19) 

vgYk.b V'ev'^ siF] 

E E cv\V',bH {Uv,b\Uv',b) 

V£Yk,b V’^N'j, g[V] 


+ E E ^v\V',bH {Sv\Ufyj^^ yyj^^,Sv^ 

V&Yk.b V'eY'i, g[V] 


K 

E E ^v\v',B 

k=l V&NK,B.k&/V'&Y'^^B[V] 


K 

= nfB+ E E (5'y|l^[l:K],[l;B]’'S’F') 


v^eVK.B F'ev'^ BiF] k=i 

where (O is due to (fT^ and (fT^ . and (l20b is due to (|4l) and (l5]l. Combining (l20l) and the induction 
hypothesis proves (fTTl) for (3 = B. ■ 


IV. Proof OF Theorem [I] 


Theorem [T] is trivially true when K = 1. The case K = 2 is a simple consequence of Lemma [3] and 
Lemma |4l Therefore, only the case K = 3 remains to he proved. 

To this end, we shall give a detailed analysis of LP^ 2 - First consider the following related linear 
program 


3 


max 

Wkrk,2 


k=l 



over rk, 2 - 

, A; G [1 : 3] , 

s.t. rk ,2 

IV 

A: G [1 : 3] 

ri,2 

+ rya > 

+ V’L.j} + V'L}, 


i,j G [1 : 3] ,i 7^ j, 
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where ipy, V G ¥ 3 ^ 2 . are non-negative real numbers. We say {cv, 2 ,V £ V 3 ^ 2 ) is an optimal Lagrange 

-- W 

multiplier of LP 3 2 if 

3 

C{k},2'^{k} + (^{i} + V’Lj} + V'L}) = , 

k=l i,j e[l:3],i<j 

CV',2 ='Wk, /c G [1 : 3] , 

V'eV3,2:fcGy 

cv',2 > 0 , L G ¥3,2, 

--'W --'W 

where denotes the optimal value of LP 3 2 - One can solve LP 3 2 with w G W 3 by considering 5 
different cases (see Table IT]). 


TABLE I 

-—w 

Linear Program LP3 2 


Case 

Condition 

Optimal Solution 

Optimal Lagrange Multiplier 

1 

w e wfl 

'*/’{!, 2 } < '*/’{!, 3} + V’{2,3} 

'0{1,3} < '*/’{!, 2 } + V'{2,3} 

V’{2,3} < V’{ 1 , 2 } + V’{1,3} 

f°P* = Ipyy + i (V'{ 1 , 2 } + V'{1.3} — i’12,3}) 
'^2^2 — i’12} + I (V’{ 1 , 2 } — V’{1.3} + V’{2,3}) 
^3^2 - i’m + 1 (-^’{ 1 , 2 } + ^>{1,3} + V'{2,3}) 

£{ 1 . 2},2 = 1 (tri + W 2 - W 3 ) 
£{1.3 },2 = \(wi-W2+ W 3 ) 
£{ 2 , 3}.2 — 1 ( — Wl +W 2 + W 3 ) 
£v .2 — 0 , otherwise 

2 

w e 

i’ll,2} > i’ll,3} + V’{2,3} 

ry2 - i’ll} + i’ll,3} 

^2^2 - i’12} + i’11,2} - i’ll,3} 

r°^2 = i’13} 

£{3 },2 = -Wl + W2 -1- W3 

£{ 1 , 2},2 — W 2 

£{1,3}, 2 =Wl-W2 

cv ,2 — 0 , otherwise 

3 

w e 

'0{l,3} > i’11,2} +i’l2,3} 

- i’ll} + i’11,2} 
r°2^2 = i’12} 

^3^2 — i’13} + V’{1.3} “ i’11,2} 

£{2}, 2 = -Wl + W 2 -1- W3 

£{1,2},2 = Wl - W 3 

£{1,3},2 — W 3 

£v ,2 — 0 , otherwise 

4 

W G W 3 

V’{2,3} > V’{ 1 , 2 } + V’{1,3} 

r°:^2 = i’ll} 

^ 2 ^ 2 * ~ i’12} + V’{ 1 , 2 } 

^3^2 - i’13} + V’{2,3} - i’11,2} 

£{1},2 = Wl — W 2 -I- W 3 

£{1,2},2 = W 2 — W 3 

£{ 2 ,3 },2 — W 3 
£v ,2 — 0 , otherwise 

5 

W G 

'0{2,3} < i’11,2} + V'{1,3} 

>" 1:2 = V'ti} 

^ 2^2 - i’12} + i’11,2} 

r°^2 - i’13} + i’ll,3} 

£{ 1},2 = Wl — W 2 — W3 

£{ 1 , 2},2 — W 2 

£{1,3},2 — W 3 

£v ,2 — 0 , otherwise 
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ij{k} = H {Uk,2\U,„2), fee [1:3], 

(^*-2, Uj^2) -il^i- V’j, 0} , z, j G [1 : 3], 1 7^ j, 

where Uk is a maximizer of maXj,g[i, 3 ]\{;j} H {Uk, 2 \U^^ 2 )- Moreover, define 

^ 3,2 = {{rk,2,k G [1 : 3]) : > 'ip{k},k G [1 : 3] , 

n ,2 + rj ,2 > V^li} + + '^{j},hj G [1 : 3], z 7 ^ i} . 

One can prove via direct verification that 7^3 2 coincides with '^ 3 , 2 - 
Lemma 5: 7 ^ 3,2 = 7 ^ 3 , 2 - 

See Fig. |2] for illustrations of 7^3 2 (i.e., '^ 3 , 2 )^ where the optimal solutions in Table jl] are highlighted. 


7?3 


i?3 



l/’l2 < '013 + '023 
013 < 012 + 023 
023 < 012 + 013 


012 > 013 + 023 


7?3 ^3 



Fig. 2. Illustrations of 7?.3,2 (i.e., 7 ^ 3 , 2 ) and optimal solutions in Table U] 
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Lemma 6: Let z, j, k be three distinct integers in [1:3]. 


(1) ^{i} + ^ 


H iU,,2,Uj,2) , 


H Uj^2) > 1p{i} + 'lp{j}, 


H{U,,2\Uk,2) + H iUj,2\Uk,2), H {Ui,2, Uj,2) < 

(2) If + V'L.fc}, then V'ljj + + V’L) = H {Ui^2, Uj^2)- 

(3) If + V'O} = H {Ui^2\Uk,2) + iUj^2\Uk,2), then 


'^{i} + + i^{k} = H {Ui^2, Uk,2) , 

'^{j} + '^{k} = H {Uj^2, Ukfi) ■ 

Proof: See Appendix lAl ■ 

According to Lemma[6l we have the following four cases for + (fj € [1 : 3] , z 7^ j): 

(Case A) 


V’ll} + V'{1,2} + V’{2} = H (Cl, 2 , 112 , 2 ) , 

V’ll} + ^{1,3} + V’{3} = H (Cl, 2 , C3,2) , 

V’{2} + ^{2,3} + V’{3} = H (C2,2, C3,2) ; 

(Case B) 


^{1} + V'{1,2} + ^{2} = H (Cl, 2 , C2,2) , 

^{1} + ^{1,3} + ^{3} = H (Cl, 2 , Us, 2 ) , 

V'{2} + V’{2,3} + '4’{3} = H (C2,2|Ci,2) + H (C3,2|Ci,2) > H (C2,2, Us,2) 

(Case C) 


^{1} + ^{1,2} + ^{2} = H (Cl, 2 , C2,2) , 

^{1} ^{1>3} + '^{3} = -ff (Cl,2|C2,2) + H (C3,2|C2,2) > H (Cl, 2 , Us,2) , 

'^{2} + V’{2,3} + '^{3} = H {^2,2, Us,2) ] 

(Case D) 


f>{l} + V’ll, 2 } + V’{2} = H (Ci,2|C3,2) + H {U2,2\Us,2) > H (Ci,2, C2,2) , 
V’ll} + V’ll,3} + V’13} = H (Ci,2, Us,2) , 

■012} + V’12,3} + V’13} = H (C2,2, Us,2) ■ 
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Now one can readily solve LP ^2 with w G W 3 by considering all possible combinations of these four 
cases and those in Table U (i.e., Cases 1-5). For example, consider the scenario where Case 2 and Case 
C are simultaneously satisfied (henceforth called Case 2C). It can be verified that 

= H {Ui,2\U2,2) , 

^{ 3 } = H {U 3 , 2 \U 2 , 2 ) , 

V’{1,3} = 0, 

V’ll} + V’{1,2} + '^{2} = H (C/1,2, C/2,2) • 

For {r'^ 2 ^ C: € [1 : 3]^ in Table H we have 

= ^{1} + V'{1,3} 

= i/(C/1,2 1 c/2,2), 

= ^{2} + ^{1,2} - V'{1,3} 

= (^{1} + V’{1,2} + V'{2}) - (^{1} + V'fl.S}) 

= //(C/2,2), 

= V'iS} 

= //(C/3,2 1 C/2,2). 

In view of Lemma [51 /c G [1 : with — ^‘k’ 2 ' k ^[1 : K], is an optimal solution of LP^ 2 - 

Therefore, the optimal value of LP ^2 is given by 

= WiH (C/i,2|C/2,2) + W2H (C/2,2) + W3H (C/ 3 , 2 |C/ 2 , 2 ) • 

Moreover, {cv\v', 2 ^ ^ ^ ^ 3 , 2 , V' G V 3 2 [f^]) with 

C{i}|{2},2 = m - ^2, 

C{3}|{2},2 - WI3, 

C{ 1 , 2 }| 0,2 - W^ 2 , 
cv\v ',2 — 0 , otherwise, 
is an optimal Lagrange multiplier of LP^ 2 . 

One can obtain the following lemma by analyzing the other combinations in the same manner. It is 
worth mentioning that not all combinations are possible. Specifically, Cases 2D, 3C, and 4B violafe 
Lemma [6t2), so such combinations are void. 
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TABLE II 

Optimal Lagrange Multipliers of LP3,2 eor All Possible Cases 


Case 

C{1}|{^1}.2 

C{2}|{i'2},2 

C{3}t{^3}.2 

C{1.2}|0.2 

C{1,3}|0,2 

C{2,3}|0,2 

lA 

0 

0 

0 

5 {Wl +W2— W3) 

1 {Wl —W2+ W3) 

1 {-Wl +W2 + W3) 

IB 

0 

1 {—Wl +W2+ W3) 

2 (—Wl +W2+ W3) 

i (Wl +W2— W3) 

i {Wl —W2+ W3) 

0 

1C 

i {wi —W2+ W3) 

0 

i (wi — W2 -1- W3) 

i (Wl +W2— W3) 

0 

i {-Wl +W2+ W3) 

ID 

2 {wi +W2 - W3) 

i {Wl +W2 — W3) 

0 

0 

i {Wl —W2+ W3) 

i {-Wl +W2+ W3) 

2A 

0 

0 

—Wl +W2+W3 

W2 

Wl — W2 

0 

2B 

0 

0 

—Wl + W2 + W3 

W2 

Wl — W2 

0 

2C 

Wl — W2 

0 

W3 

W2 

0 

0 

3A 

0 

— Wl +W2+W3 

0 

Wl — W3 

W3 

0 

3B 

0 

— Wl + W2 +W3 

0 

Wl — W3 

W3 

0 

3D 

Wl — W3 

W2 

0 

0 

W3 

0 

4A 

Wl — W2 + W3 

0 

0 

W2 — W3 

0 

W3 

4C 

Wl — W2 + W3 

0 

0 

W2 — W3 

0 

W3 

4D 

Wl 

W2 — W3 

0 

0 

0 

W3 

5A 

Wl — W2 — W3 

0 

0 

W2 

W3 

0 

5B 

Wl — W2 

0 

W3 

W2 

0 

0 

5C 

Wl — W2 — W3 

0 

0 

W2 

W3 

0 

5D 

Wl — W3 

W2 

0 

0 

W3 

0 


Lemma 7: For linear program LP^2 with w € W 3 , (cy|v'',2) ^ € ¥3^2; 2 I^]) Table III is 

an optimal Lagrange multiplier. The general case w G can be reduced to the case w G W 3 via 
suitable relabelling. 

The next result shows that ® holds when K = 3 , which, together with ([ 3 ]) and Lemma IH completes 
the proof of Theorem [T] 

Lemma 8: Let {cy^i,V G ¥3^1), (cy|v'',2)¥ G ¥3^2;^' G ¥32 [¥]), and (cy^3,¥ G ¥3^3) be the opti¬ 
mal multipliers in Lemma [U Lemma | 2 j and Lemma | 7 j respectively. We have 

cv,iH{Xv)> Y. Y. cy\y,^2H{Xv\Xv') ( 21 ) 

> Cy,sH {Xy\X[y3]\y) ( 22 ) 

y £^ 3,3 

for all X[i.3]. 

^We set C{fe}|{fc /},2 = 0 for k’ / at- 
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Proof: Note that (|2TI) follows from Lemma |3] The proof of (l22l) is relegated to Appendix |B] ■ 

V. Proof OF Theorem [2] 

The proof of Theorem [2] also largely follows the general approach outlined in Section |III1 However, 
due to the symmetry assumption, some simplifications are possible. 


A. Linear Program 

When the distrihution of Uyi.K]\i-.K] is symmetrical entropy-wise, H {Uv^aWv,a) depends on L € 
Vk,oi and V' € [L] only through |1/|; for this reason, we shall denote it as and rewrite 

LP^ „ in the following simpler form 

K 

>-Px,a : min ^ Wkrk,a 

k=l 

over rk^a, k^[l: K], 

S.t. ^ y 

kev 

Definition 3: We say {cv,a,V G V/r,a) is an optimal Lagrange multiplier of with w G if 

Cv,aHlv\,a = ( 23 ) 

V^Vk.c 

cv,a = Wk, k e[l: K], (24) 

VGVK.c-kev 

Cv,a >0, V e YK,a, (25) 


where /„ denotes the optimal value of LP^ 
For f € [0 ; a — 1], define 


.(0 A 
' k,a ~ 


Hk.a kii 


fc —1,0:5 


a—I ’ 


k 

k € [I+ 1:K], 


where Tfo,Q — 0- 

Lemma 9: {j^k^,k G [1 : G TZK,a- 

Proof: See Appendix ICl ■ 

For a G [1 : Ff] and f G [0 : a — 1], define 


Ll^l^^{V<Z[l-.K]-.\V\=a,[l-.l]<ZV}. 
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Recall that ■ ■ ■ , form a partition of Wk- For w G Wk and a G [1 : K], let denote 

the unique integer in [0 ; a — 1] such that w G it is easy to verify that 

0 = lY <IJ <■■■< Ik-1 < Ik- 
Moreover, for w G Wi^- and a G [1 : iF], define 


\w A 


a-ll 


K 

E 


Wk 


and 


C W A 

— 


{(cv,a ■ V G ^K,a) ■ 


^[l:k],a 

Wk Wk-\-l^ 

ke[i-.1:^-1], 

(26) 

qi:/w],o = 

-- - A^, 


(27) 

CV,a > 0, 

V G 


(28) 

CV,a = 0, 

Otherwise, 


(29) 

E 

CV,a = Wk, 

A: G + 1 : K]}. 

(30) 





Note that (l26l) and (|27]) are void when = 0. The definition of can he extended to the case 
w G through suitable lahelling. 

Lemma 10: For any w G Wk, icv,a-i,y € G icv,a,y G Vi^,a) G 


where 


^ CV,a = K, 

cv,a = Wk, A; G [1 : iT] , 

v^VK.c-kev 

Cv,a >0, 1/ G VK,a, 




y~l CF,a = 

VGVk,« 


K 

I E 

k=l 

Wl, 


C>0, 
C = o, 
C>o. 



£ (« 


l- k) C[i,fc] 



1 

a — 1 — / 


W 

a—1 


(31) 

(32) 

(33) 

(34) 

(35) 

(36) 
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Proof: See Appendix iDl ■ 

The main result of Section IV-AI is as follows. 

Lemma 11: For linear program with w G A: G [1 : AT]^ is an optimal solution, and 

every {cv,a, V G '^K,a) G is an optimal Lagrange multiplier. 

Proof: In view of Lemma|9j we have , A: G [1 : AT]^ G lZK,a- Consider an arbitrary {cv,a, C G YK,a) € 
It follows from Lemma [TOl that (cy^a^C G YK,a) satisfies (|2^ and (l25l) . Note that 


K 


^ ='^Wk {Hk^a - Hk-l,a) + ^ 


K 


Wk- 


Ha,a Hi™,a 


k=l 


k=l 


k=l™+l 


a-ll 


{wk — Wk+l) Hk,a + {wi™ - A^) Hi-^^a + ^aHa,c 


k=l 

C -1 


^ ^ “h '^[1.1™],qHi™, a T ^ ^ C.y^ciHa,c 


k=l 




(C) 


(37) 


veNK,c 

where (iTTl) is due to (ISTI) . On the other hand, for any (r^ „ : A: G [1 ; A']) G lZK,a, 

K K 

^ ^ '^k'^k,a ^ ^ 'y ^ ^V,a'l"k,a 

k=l k=lV&Y K,c,'-k&V 

— y ^ Cy,« y ^ f’k,a 

VgYk.o, k£V 

> ^ CV,a7A|y|_o. 

VGYk.c, 

Therefore, \ A: G [1 : is an optimal solution. This also shows that {cv,a,y S YK,a) satisfies 

1), fhus is indeed an optimal Lagrange mulfiplier. ■ 


B. Entropy Inequality 

Define fhe indicator function 


Z (event) = < 


1, event is true, 
0, event is false. 
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Lemma 12: For w G Wk with > 0 and {cv,a, V F ^K,a) G define {cv>,a-i, V S 'VK,a-i) 

as follows: 


C[l:fc],a -1 -Wk- Wk+ 1 , k€ [l: - 1] , 




A \ w 

— 1 1 ’ 


CV',a-l — 


^ + ^ X X X 


\w / ^ ' \v 


ven}^%f:V'cv 


r=k-\-l 


V gQ 


(C-0 

K,a— 1 ^ 


cv',a-i — 0, Otherwise, 


where 0^ is given hy (l36l) . The following statements are true. 

(1) (cv',a-l,V' G VK,a-l) G 

(2) We have 


X CV',a-lH {Xv) > X ^V,aH (Xv) 

V'€Yk,c-i VgVk.c 


for all X^i-xy 

Proof: See Appendix IE] ■ 

Lemma 13: For any w G there exist {cv,a, E G YK,a) G a G [1 : K], such that 


X cv',a'H{Xv>)> X <^V,aH{Xv) (38) 

V'GVk-.c' VGYk,c 

for all Xyi-x] and a > a'. 

Proof: By symmetry, it suffices to consider w G Wx- We shall first assume wk > 0, which implies 
Aq > 0, a G [1 : K]. Define {cv,k-, V G Yk,k) with 


C[i:k],K - Wk - Wk+i, A: G [1 : AT] , 
cv,K — 0, otherwise. 


It is easy to verify that {cv,k, Y G Yk,k) G One can successively construct the desired (c\/,a, V G Vx,a) 

from a = A' — ltOQ: = lhy invoking Lemma \V2\ 

Now consider the case roi > • • • > wk-i > wk = 0. The preceding argument implies the existence 
of (oy,Q:) E G YK,a^ G a G [1 : AT - 1], such that 

X c'y,^^,H{Xv>)> X ^v,aH{Xv) 

V'GVk--i,„' V£Yk-i,c 
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for all and a > a', where w' = (wi, ■ ■ ■ , Define (cy^o, 1^ G ^K,a), a G [1 : if' — 1], 

with 


cv,„ ^ 4 ,,, K^V, 
cv,a — 0, Otherwise, 

and {cv,K,y G Yk,k) with 

cv,K = c'yk-i, K ^ V, 
cv,K — 0, otherwise. 


It is easy to verify that such {cv,a, V G YK,a), a G [1 : K], have the desired properties. The genera 
where rui > • • • > wk'-i > WK' = • ■ • = wk = 0 for some K' < K can he handled via induction 


. case 

i ■ 


C. Outer Bound 

The following result, together with ([3]) and Lemma [T3l completes the proof of Theorem |2] 

Lemma 14: If any w G there exist (cy^Q,I^ G ^k, a) G a G [1 : iT], such that (l3^ holds, 

then 




when the distrihution of [i,^] is symmetrical entropy-wise. 

Proof: Let {Rk : A: G [1 : iT]) he an arbitrary admissible rate tuple. It suffices to show that 

K K 

^WkRk>^7a- (39) 

k=l a=l 


Without loss of generality, we assume w ^Wk- We shall prove via induction that, when the distribution 
of t/[i ■.K],[i-.K] is symmetrical entropy-wise, for any D-MLDC system satisfying ([Til and ([21), 

K /3 

'^Wk{Rk + y) >'^fa + ~'^^[l-.k],l3H (^S[i,k]\U\l:KUl:l3]^^\k+l:/3],K) 
k=l a=l k=l 

1 ^ 

+ - cv,ftH (^Sv\U^Y.K],m) - PSe^Wk, /3G[l:iT], (40) 

” I'lw'i k = l 

where Sg tends to zero as e —0. One can deduce (l39l ) from (l40b by setting (5 = K and sending e —0. 


®If u;i = • • • = Wk = 0, then cv,a = 0 for all V € Vk.c and a £ [1 ■. K], 
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The proof of (l40l) for /3 = 1 is the same as that of (ITT]) . Now assume that (l40l) holds for /3 = i? — 1. 
In view of (l3^ . we have 


cv',b-iH (^Sv'\U^, 




V'&n), 




k=l 




+ ^ cv,bH 

-,{'b) 




(41) 


veoi 


where — c^i-.k],B-i^ {k < ^b-i} > 0, /c G [1 ; Z^], according to (1^ . (1^ . and the fact that 

C[i:fc],B-i = C[i:fc],B when ke[l: l^-i - l] ■ Moreover, 

- C[l-.k],B-l^ {k < ^B-l}) H 


k=l 

^B 

- X] ~ C[l:k],B-l^ {k < Ib-i}) ki ('S'[l:fc]|t^[l:K]Jl:S_l]) 

fc=l 

^B ^ 

^ n 


— [1:5-1]) 
k=l 

^B-1 

k=l 

II 

— (^'S'flifcj |t^[l:j:r],[1:5-1]) ^[fc+l:5],[B:X] 
fc=l 

^B -1 


y~! c[i.fc]^^_iB 


fe=i 


:K],[1:5-1] ’ ^[fc+l:5-l],[5-l:i<'] 1 ’ 


(42) 


Combining (HTI) and (|4^ gives 


*'B -1 

^ C[i.fc]_5_lB ^S'[l.fc]|C/|-”.;:^] ^ Cv'',5-l-fZ^ S’!/'|t^[l:X],[1:5-1]) 


A:=l 




> 


^ C[i.fc]^5B ^S'[l.fc]|C/j’];.^] [^.^]) + ^ (^<S'y|f^Ji:5']Ji:5_i]) • (43) 


k=l 
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Note that 

H {S[i..k]\U^, 

= H {S]^i,B]\U^i,K],[l-.B-l]^^[k+l-.B],[B:K]^ ^[k+l:B] ^ 

— ^ (^^ll:B],[l:B]^ ^[l-B]\U^:K],[l:B-l]^^lk+l:B],[B:K]^ ^[k+l:B] ^ 

— ^ (y^[l-.B],[l:B]^ ^[l-.B]\U^:K],[l:B-l]y^[k+l:B],[B:K]^ ^[k+l-.B]^ — nd^ 

— ^ (j^[l:B],[l:B]\^[l:K],[l-.B-l]^^\k+l:B],[B-.K]^ ^[k+l:B]^ 

+ H U^:B],[1:B]^ ^\k+l:B],[B:K]^ ^[k+l:B]^ “ n6e 

— ^ {y^:B],[l:B]\^[l:K],[l-.B-lY ^[k+l:B],[B:K^ 

+ H (^S[i:k]\U[i.K]^[l:B-l]^U\l:B],[l:B]^^\k+l-.B],[B:K]^ ~ 

> nHk,B + H - nJe, k e [I : Ib] , 

where (1441) follows hy Q and Fano’s inequality. Similarly, we have 


(44) 


(45) 


H 


1 :K],[ 1 :B- 1 ] 




[l:ir],[l:B-l]) ^ (^y,[l:a]l^[l:X],[l:B-l]> 


> H 


^V,[l:B]^ ^v\Ull-,K],[l:B-l]) 




> ''t^H\v\,b 


+ H 


[l:ir],[l:B-l]j ^V,[ 1 :B] 

■)(^b) 


— nSf 




(46) 
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Continuing from (l43l) . 


"B-l 


fc=l 


'' *=“k,b-i 


‘B 


veVK,, 


+ ^ cv,bH 


k=l 


n 

1:KU1:B] 




nSe ^ 
V£Vk,b 


(47) 


> ^ (‘^[l:fc]l^[l:ir],[l:B]) ^[fc+l:S],[B:ir] 


k=l 


+ ^ cv,bH 

,{'S) 


if 


[l:ir],[l:B] 


n6e'^Wk, 


(48) 


veni 


k=l 


where (l47l) is due to (l45l) and (|4^ . and (|4^ is due to (|2^ . (1351) as well as Lemma [TT] Combining (1481) 
and the induction hypothesis proves (l40l) for fi = B. ■ 


VI. Conclusion 

We have characterized the admissible rate region of D-MLDC for the case K < 3 and the case where 
the source distribution is symmetrical entropy-wise. In view of the intimate connection between MLDC 
and its lossy counterpart known as multiple description coding llTdl . it is expected that the results in the 
present work may shed new light on the robust distributed source coding problem (which is the lossy 
counterpart of D-MLDC) studied in ITSl . 


Appendix A 
Proof of Lemma [6 ] 

Proof of Part (1) of Lemma It follows by the definition of V'fiy} that 

'^{i} + V’LJ} + ^{j} = H t^i.2) 

when H (74,2, t7j, 2 ) > i>{i} + V’{j}- 

When H (77i,2, 7/j,2) < f{i} + '>P{j}, we must have 
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and consequently 


i’{i} + + V'o} — H {Ui^2\Uk,2) + H {Uj^2\Uk,2) ■ 


Proof of Part (2) of Lemma^ Note that V'liy} > '4’{i,k} implies V’lij} > 0- H then follows 

by the definition of that 


V'i*} + i’iij} + = H {Ui^2, Uj^2) ■ 


Proof of Part (3) of Lemma^ In view of the fact that '(/’{i} > H {Ui^ 2 \Uk, 2 ), V’{j} > H {Uj^ 2 \Uk, 2 ), 
and V’lij} ^ 0 > we must have 

f{i^ = H{Ui,2\Uk,2). (49) 

V'{,} = F(C/,- 2 |C/fc, 2 ), 

V’hJ} = 0 

when V’{i} + 'i/’fij} + i’ij} = H {Ui^ 2 \Uk, 2 ) + H {Uj^ 2 \Uk, 2 )- Note that 

H{U,,2, Uk,2) = H{U,,2\Uk,2) + H{Uk,2) 

= + H{Uk,2) (50) 

> ip{i} + f{k}, 

where (l50l ) is due to (l49l) . It follows by symmetry that H {Uj^ 2 ,Ukfi) > '^{j} + f’lk}- Invoking the 
definition of nnd completes the proof of Lemma ■ 

Appendix B 

Proof of (1221) in Lemma [8 ] 

It suffices fo consider w € W 3 . 
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Case lA (w € W^^): 

^ {Wi +W2- W3) H {Xi,X2) + ]^{wi - W2 + W3) H {Xi,X3) + ^ (-Wi +W2+ w^)H {X2,X^) 

= {wi - W2) {H (Ai, X2) + H (Ai, X3) - H (Ai, X2, As)) + {w2 - W3) H (Ai, A2) 

+ ^ (-«;i + W2 + W3) {H (Ai, A2) + H (Ai, As) + H (A2, As)) + (w^ - W2) H (Ai, A2, As) 

> (w;i - W2) H (Ai) + {W2 - W3) H (Ai, A2) 

+ \ i-wi + W2 + ws) {H (Ai, A2) + H (Ai, As) + H (A2, As)) + {wi - W2) H (Ai, A2, As) 

> {Wi-W2)H{X3) + {W2-W3)H{XI,X2)+W3H{X,,X2,X3) ( 51 ) 

> {wi - W2) H (AllA2, As) + {W2 - W3) H (Ai, A2IA3) + W3H (Ai, A2, As), 

where dSTI) is due to Han’s inequality ifThl . 

Case IB (w E ^2 = ^3 = 1 )^ 

i {-W 3 +W2 + W3) {H (A2IA1) + H (AslAi)) 

+ ]^{wi+W2- W3) H (Ai, A2) + ]^{wi - W2 + W3) H (Ai, As) 

> ^ (-u'l + ^^2 + W3) {H (A2IA1, As) + H (AslAi, A2)) 

+ ]j^{wi+W2- W3) H (Ai, A2) + ^ (tui - u;2 + W3) H (Ai, A3) 

= (t(;i - n; 2 ) (i^ (^1, ^2) + ii" (^1, ^3) - ^ (^1, ^2, ^3)) + (^1, ^2) 

+ ^ (-rui + u;2 + n; 3 ) {H (A3IA1, A2) + H (Ai, A2)) 

+ ^ (-u'l + W2 + W3) {H (A2IA1, As) + H (Ai, As)) + (n;i - n;2) H (Ai, A2, A3) 

> (mi - m2) H (Ai) + (m2 - ms) H (Ai, A2) + W3H (Ai, A2, A3) 

> (mi - m2) H (Ai IA2, As) + (m2 - ms) H (Ai, A2I A3) + W3H (Ai, A2, A3). 
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Case 1C {vi = t's = 2): 

^ (u;i -W2 + ws) {H (X1IX2) + H (X3IX2)) + \{wi+W2- W3) H (Xi, X2) 

+ ^ (-^1 + W2 + W3) H (X2, X3) 

>^{wi-W2 + ws) {H (X1IX2, X3) + H (X3IX1, X2)) + ^ ( 7 /;i + 7/;2 - ti^s) H (Xi, X2) 
+ ^ (-'“^1 + ^^2 + W3) H (X2, X3) 

= K - u;2) ^ (X1IX2, X3) + {w 2 -W 3 )H (Xi, X2) 

+ ^ {-wi + W2 + ws) {H (X1IX2, X3) + H (X2, X3)) 

+ ^ (U'l - ^^2 + W3) {H (X3IX1, X2) + H (Xi,X2)) 

= (u;i - W2) H (Xi 1X2, X3) + {W2 - W3) H (Xi, X2) + W3H (Xi, X2, X3) 

> (u;i - u;2) ^ (Xi 1X2, X3) + {w2-W3)H (Xi, X2IX3) + W3H (Xi, X2, X3). 

Case ID {ui = 1/2 = 3): 

i (7/;i +W2- W3) {H (X1IX3) + H (X2IX3)) + ]^{wi-W2 + W3) H (Xi, X3) 

+ ^ (-'“^1 + ^^2 + W3) H (X2, X3) 

>l{wi + W2- W3) {H (Xi|X2,X3) + H (X2IX3)) + \{wi-W2 + W3) H (Xi,X3) 

+ ^ (- 7 (;i + W2 + W3) H (X2, X3) 

> (u;i - W2) H (X1IX2, X3) + {W2 - W3) H (Xi, X2IX3) 

+ \ {-wi + W2 + W3) {H (X1IX2, X3) + H (X2, X3)) 

+ ^{wi-W2+ W3) (F(X 2 |Xi,X 3 ) + F(Xi,X 3 )) 

= {wi - W2) H (Xi 1X2, X3) + iw2 - W3) H (Xi, X21X3) + W3H (Xi, X2, X3). 
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Cases 2A and 2B (w G 

(-n;i + W 2 + ws) H {Xs\X,,) + W2H (Ai, X2) + {w^ - W2) H (Ai, X3) 

> (-ini +W2 + wz) F (X3IX1, A2) + W2H (Ai, X2) + {wi-W2)H (Ai, X3) 

= {wi-W2){H{Xi,X2)+H{Xi,X^)-H{Xi,X2,X^)) + {w2-w^)H{Xi,X2) 

+ {-wi +W2 + W3) {H (X3IX1, X2) + H (Ai, X2)) + (n;i - W2) H (Xi, X2, X3) 

> (n;i - n;2) H {X^) + {w2 - W3) H (Ai, X2) + W3H (Xi, X2, X3) 

> (n;i - n;2) H (XiIX2, X3) + (n;2 - 1^3) H (Xi, X2IX3) + W3H (Xi, X2, X3). 
Cases 2C and 5B: 

(n;i -W2)H (Xi|X,J + W3H {X3\X,^) + W2H (Xi, X2) 

> (n;i-zn2)i/(Xi|X2,X3) + n;3i^(X3|Xi,X2) + zn2i^(2Ci,X2) 

= (n;i - W2) H (X1IX2, X3) + (n;2 - W3) H (Xi, X2) + W3H (Xi, X2, X3) 

> (n;i - W2) H (X1IX2, X3) + (n;2 - W3) H (Xi, X2IX3) + W3H (Xi, X2, X3). 
Cases 3A and 3B (w G 

{-wi +W2 + W3) H (X2|X,J + {wi - W3) H (Xi, X2) + W3H (Xi, X3) 

> (-n;i + n;2 + n;3) (^2 | 2 fi, X3) + (n;i - n;3) F (Xi, X2) + W3H (Xi, X3) 

= (n;i - n;2) {H (Xi, X2) + H (Xi, X3) - H (Xi, X2, X3)) + {w2 - W3) H (Xi, X2) 
+ {wi - W2) H (Xi, X2, X3) + {-wi +W2 + W3) {H (X2IX1, X3) + H (Xi, X3)) 

> (n;i - W2) H (Xi) + {W2 - W3) H (Xi, X2) + W3H (Xi, X2, X3) 

> (n;i - W2) H (Xi 1X2, X3) + {W2 - W3) H (Xi, X2IX3) + W3H (Xi, X2, X3). 
Cases 3D and 5D {vi = 122 = 3): 

{W3 - W3) H (Xi 1X3) + W2H (X21X3) + W3H (Xi, X3) 

> (n;i - W3) H (X1IX3) + W2H (X2IX1, X3) + W3H (Xi, X3) 

= (n;i - W2) H (Xi 1X3) + {W2 - W3) H (Xi, X2IX3) + W3H (Xi, X2, X3) 

> (n;i - W2) H (X1IX2, X3) + {W2 - W3) H (Xi, X2IX3) + W3H (Xi, X2, X3). 
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Cases 4A and 4C: 

(n;i -W 2 + w^) H {Xi + {w 2 - n^s) H{Xi,X 2 ) + W 3 H (X 2 , X 3 ) 

> (n;i -W2 + W3) H {X, 1 ^ 2 , X 3 ) + {w2 - W3) H (Xi, X 2 ) + W3H {X2, X3) 

= {wi - W2) H (X1IX2, A3) + {W2 - W3) H (Ai, A2) + W3H (Ai, As, A3) 
>{wi-W2)H (AllAs, A3) + {W2 - W3) H (Ai, A2IA3) + W3H (Ai, A2, A3). 

Case 4D {vi = 1/2 = 3): 

wiH (AllA3) + {W2 - W3) H (A2IA3) + W3H (As, A3) 

> (n;i -W2 + W3) H (Ai| As, A3) + {w2 - W3) {H (A1IA3) + H (Asl A3)) + W3H (As, A3) 

= {wi - W2) H (AllAs, A3) + {W2 - W3) {H (AllAs) + H (AslA3)) + W3H (Ai, As, A3) 

> {wi - W2) H (All As, As) + {W2 - W3) H (Ai, A2IA3) + W3H (Ai, As, A3). 

Cases 5A and 5C: 

{W3 -W2- W3) H (Ai I A,,) + W2H (Ai, As) + W3H (Ai, A3) 

>{w3-W2)H (Ai I As, As) + {w2 - W3) H {X3,X2) + W3{H {Xi, X2) + H {X,, X3) - H (Ai)) 
>{w3-W2)H ( Ai I As , A3) + (n;2 - n^s) ^ (^1, ^2) + W3H {X, , As, A3) 

> {W3 - W2) H (AllAs, As) + {W2 - W3) H (Ai, A2IA3) + W3H (Ai, As, A3). 

Appendix C 

Proof of Lemma [9] 

The following result is needed for the proof of Lemma |9] 

Lemma 15: Assume that H (Xy) = H {Xv) for all V, F' C [1 : K] with \V\ = \V'\. We have 

H (A[i,^^]|A[^^_n,^-]) ^ H (A:[i:t,]|A[^,^+i,j]) 
k ~ k 

for any kk 2 ,j € [1 : K] such that ii < is < J- 

Proof: It suffices to consider the case ii < is- Note that 

*2 

kH (Aiji:*^]IA[j 2 _|_i,jj) = (Aji-j^jIA[j 2 _,_i.j]) 

k=l 

*2 L 

k=l k=l 

> H (A[i,i,^]|A[i,^+i,j-]) +kH (A[i,i,^_i]|A[j^,j]) . 
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Therefore, 

i2 - 1 “ ^2 


One can readily complete the proof via induction. ■ 

Now we are ready to prove Lemma |9] 

Proof of Lemma^ Consider an arbitrary V G ^K,a- Let Vi = Cn [1 : Z] and V 2 = C\Vi. It suffices 
to show that 

H ([/fc,a|C/[fc+l:„],«) + |^2| ^ ^ ■ (52) 

keVi 

First consider the case Cj / 0, f = 1,2. Note that 


|L| 

^ {^k,a\U[k+l:a],a) — ^ I f2[(Vi)^+l:Q] ,a) 


fceL 

r=l 



ILI 



^ (6^T,a 

T=1 

(53) 


= H {Uli.\v^\la\U[\V^\+l-.a],a) , 

(54) 


where (15^ is due to the fact that (Vi).^ > r for r € [1 : |l/i|] and that the source distrihution is symmetrical 
entropy-wise. Moreover, we have 


l^2| 


Lf (U[l-irl:a],a) 

a — I 


^ Lf {U[l-.a-l],a\U[Q-l+l:a-\Vi\],a) 

~ a — I 

> H (t/[l:|F2|],a|C2[|F2| + l:a-|yi|],a) 

= H {U^\Vi\+l:\V\],a\U[\V\+l:a],a) , 


(55) 

(56) 


where (l55l) is due to Lemma [15] and the fact that a — I > | V 2 1 ■ Combining (l54b and (l56l) proves ([52]) for 
the case Cj / 0, i = 1,2. 

Note that (l52l) degenerates to (l54l) when V 2 = 0 and degenerates to (l56l) when Vi = 0. This completes 
the proof of Lemma jH ■ 
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Appendix D 
Proof of Lemma [Tol 


Proof of 07]) ; Note that 


K 


K 


Y. Y. Y 

fc=zw+i ^='S'+VGO^“Uey 

= Y Y ^ 


yeo.. 
= (a — 


(is) fesyMi:;”] 


C) E 

Os) 




from which the desired result follows immediately. 
Proof of 02]) ; Consider the following two cases. 
(Case l)k^[l-. l'^]: 


Yh = E + E 

VeYK.c-k&V i=k 




Wk- + E 

,0s) 




= Wk, 


where (l57]) is due to dST]) . 
(Case 2j A: E + 1 : K]: 


E = E 


yGo(‘”j:.Gy 


(57) 


Proof of 07]) ; It suffices to verify that ru;™ — > 0 when /^ > 1. Indeed, this is a simple 

consequence of the fact that w E W^“^. ■ 

Proof of 0?]) ; Consider the following three cases. 

(Case 1) /q -1 <la — Note that 

C C 

E (a-l-k) = (a - 1 - C-i) w^C-i+i “ E - (« - 1 - C) (58) 
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and 


C K 

fc=/^_i+l fc=/^ + l + 1 

K 

= ^ Wk 

fe=C-i+i 


= (o-l-C-i)A^i. 


We have 

C = j - XI (a - 1 - ^) C[l:fc],, 




— (a — 1 — C-i) + X Wk + {a — 1 — 1^) XI 


(o-C)A'+ "£ U.I - (o - 1 - C-i) ”>C-.+i 


o-l-C-1 


A:=/^_i + l 


a - 1 - '?-i 


= {(« - 1 - C-.) - (o - 1 - C-i) ,w 

'■a—1 

= A^_i - 


where (l60l) and (IM1) are due to (l5^ and (l5^ . respectively. It can he verified that 

K 

(o-i-c-i)A^i= wt 

fc=i*_j+i 

K 

= Wk + Wl~_^ + l 

A:=i”_i+2 

> (a - 2 - C-i) + i 

= [a-1- wi^_^+i, 

where (l6^ follows from the fact that w € Combining (1^ and (IMl) proves 0^ > 0 


(59) 


(60) 


(61) 

(62) 


(63) 

(64) 
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(Case 2) 


We have 


— 1: Note that 

K 

{a-C)Ar= ^ w, 

k=l^^l 


= (a - 1 - C-i) A^-i - lOi; 
= (a-C)A~_,-io,;. 


C = (AJ - (a - 1 - C) C|i„;],„) 

= {A--{a-l-C) K-A:))^ 

CK 6 q, 

\ W 

= A„_i - wi~ 




where (1^ is due to (1^ . The faet that 0^" ^ 0 follows hy (1^ and (1^ . 
(Case 3) Note that 

1 




Moreover, we have 


\W _ 

^r\ 


a - 1 - C-i “ (a - 1 - l'^) (a - l'^) 


K 

K 


a-l-ly3 a-ll 


E 


k=i'~+i 


_ \ w \ w 

— "^a-1 “ ■^a 




Proof of diJ]); Consider the following two cases. 
(Case 1) = 0: 


Yj = Yj 

V&Yk,o. 


1 

a 


K 

XI W'fc, 

k=l 


(65) 


( 66 ) 


(67) 
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where (l67]) is due to (l3TI) . 
(Case 2) > 0: 


K-l 


'y ^ ^V,a ^ ^ ^[1:1™],a ^ ^ 


CV,c 


VeVK.a 


where (IMI) is due to (l3TI) . 


k=l 

tr-1 




(Wk - Wk+l) + (wi^ - A^) + ^ Cv,a 


k=l 

C-1 


veQl 


^ (wk - Wk+i) + (wi^ - Ao) + A 


k=l 

= m, 


( 68 ) 


Appendix E 
Proof of Lemma O 

Proof of Part (1) of Lemma \T2\ Note that (l26l ). (ITTl ). and (l29l ) obviously hold. Moreover, (|28] ) is 
implied hy (l34l) . Therefore, it suffices to verify (l30l) . 

Consider an arbitrary integer k € [l^-i + 1 : iT] - We have 

CV,a 


E E 

V'en^l-f):k(,V' ven^Z^-.V'cv 


« - 1 - 


E 


G -kGV'CV 




a-I- 


cy,c 


E 


cy,c 


ven^K^J-.kev 


Note that 


E 

(C i) 


W E ""Ca E H 

d's) *=C-i+i ' ^ 


v&n\ 


T=i+1 


(69) 


4 E 'I'." E E E = ™ 


v^€n 




/C,o 


r=i+l Ow ) 

V'&n)^Z._Y -.kev 
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Moreover, 


E E = 


T—i+i riw /j 

V'en)^ ^~_V :keV' 


Therefore, 


- ^ ^ (''^)[l:a]\M} 

T=i-\-l 

= {a-l-i)I{keV}+l{ke (V)[ 1 ,]} , y € € [C -1 + 1 : C] • 

^ Co 

(C-i) “ *=C-1+1 ^=*+1 

y'eQ]c“_i .fceV' 

^ C 

X '^Eo X] '^[l:*],Q ((“ “ 1 ^ ^ (^)[l:i]}) 


Fen 


('") *=C-i+i 


K,o 


^ X (“ “ 1 “ *)'^[l:j],Q X 


cy,a 


+ 


" *=C-i + l 
1 


fjw') 

VgQ^“J:/cGV^ 


q; 


X “^Eo X '^[l;*],o^|^ ^ (^)[l:i]} > 




where (1721 ) is obtained hy substituting (TtT]) into (iTOl ). Combining (|6^ and (IT^ gives 


X 


CV'a-l 


ik&V 


X ^Eo + ^ X ^Eo X C[l:*],o^{^ ^ (^)[l:i]} 

E'rv , , . ,___ ^ ^ ■' 




vm^l^J-.kev * ^"-1+1 


Now consider the following two cases. 
(Case 1) k ^ + !:/„]: We have 


X '^Eo = X '^Eo 

V£n\i^J:k£V ven^l^J 


= x: 


(71) 


(72) 


(73) 


(74) 


(75) 


and 


|/c € —Z{k^[l:i]}, V € ^ [C-i + 1 • C] 


(76) 
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where (TTSI) is due to (l3TI) . Continuing from (1741) . 


E 


CV',a-l 


i) 

0.-1 -k&v' 




ven^S^J-.kev 


ven^‘^J-.kev 


CV,a Y ^ y]} 

i=l'^_i-\-l 


1 


ven^S^J:k£V 




i=k 




i=k 


= U)k: 


where (ITT]) and (TT^ are due to (1761) and (1751) . respectively. 
(Case 2) k & [l^ + 1 : K]: We have 


E 


CV'a-l = 


E 


cy,c 


i) 

V'enk k~_i -.k^V' 


vm^‘°J:kev 


= Wk, 


where (17^ follows hy (l74l) and the fact that 

^{k^{V)[i..i]}=0, [C_i + l:C] 

This completes the verification of (l30l) . 

Proof of Part (2) of Lemma 172 } Note that 
|V| 

E 

T=i+1 

|V| N ""I / 

= E E ^K),: 

r= 2 +l r= 2 +l 

|V"I 

= E «(^(V),,,„.,IVV),„) + (|'^|--)'^K).) 

T=i+1 


Mi3V|l\{x} 


/[l:i] 


> (|y| -l-i)H J + (|C| -i)H 

= (|y| -l-i)H (Xy) + H J , i G [0 : |y| - 1] , 


(77) 


(78) 


(79) 


(80) 

(81) 


May 4, 2015 


DRAFT 


37 


where ([80l) is due to Han’s inequality lIThl . We have 


E 


a; 


CV,aH {Xv) 




V'en 


K,oc-l 


ej_ 

\ w 

/\/-V 

e: 


E 


CV,a 


E 


H(Xv) 


Fen 


K,ot 




^ E E « 


> 


a; 




^(y> 


[l:c]\{x} 


^ Cy,a((a-1-/ 


yen 




^Q;—1 




X| 


1:^:; 


y~! Cy,ai? (Xy) - — ^ (a - 7 - 1) C[i.j]_o ^ Cv,qH (Xy) 






V€n 

where ([82l) follows hy (IMl) . and ([83]) is due to (ISTl) and (l34l) . Moreover, 


ven 






E ly E E ‘:M.»Ei{'"' = (y|i:< 






]\{r} 


T=?+l 


(82) 


(83) 


*"0: <-« 

= ^ ^(^W>y.i\y}) 

“ yGn^“) i=C-i+i 

^ C 

X] X] C[i.j]_o ((a - 1 - i) i? (Xy) + (X[i.j])) (84) 

" ViT XI (« - 1 - *) C[l;i],« X (^V) 

1 

+ ^ X Cii.il'xH {^[i-.i]) X ^Eo 

^ I- I- 

'y ^ ^ 0 *"[l;i],o ^ ^ Cy^cfH (^Xy') + ^ ^ (^[l:i])) (^5) 

“ *=C-1+1 *=C-1+1 

where (IMl) and (ISSl) are due to (IMl) and (l3T]) . respectively. Now one can readily complete the proof hy 
combining (l82l) and (l85l) and invoking the fact that = C[i:i],a when i G [l : - l]. ■ 
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